Devices called encoders, located inside the sensors on autonomous cars, convert motion and other information acquired on the road into electrical signals so that can then be used as feedback for the software running the vehicle and make informed decisions.
The process is a lot like how we humans use the feedback traveling from our five senses to our brain in order to learn and proceed wisely.
But what if you have to replace a sensor, and the new part encodes information differently than the deep neural network, or DNN — the brain trained to control the car?
Well, when the replacement sensor is a different brand or some other mismatch, that’s a safety concern, according to Meriel von Stein, a Copenhaver Fellow at the University of Virginia School of Engineering and Applied Science.
“The DNN can no longer reliably infer information from the new sensor hardware,” von Stein said. “This can happen with updates to the version of autonomous system, but it can also happen during aftermarket swapping out of one sensor for another.”
To help solve the problem, von Stein invented an augmented encoder called PreFixer. She published her solution in the July issue of IEEE Robotics and Automation Letters, which coincided with her successful dissertation defense over the summer.
LESS Lab Accepts the Challenge
Von Stein was advised by Sebastian Elbaum, the professor of computer science who leads UVA Engineering’s LESS Lab (Leading Engineering for Safe Hardware), of which von Stein is a member. She authored her paper on PreFixer’s proof of concept in collaboration with Hongning Wang, a visiting associate professor, and Elbaum.
Elbaum’s software systems lab prides itself on mentoring researchers and taking on challenges that include “learned components and interactions with the physical world.”
Von Stein is a former software engineer at NASA Goddard Space Flight Center and NASA Kennedy Space Center with a long track record of investigating and solving autonomous systems problems in both vehicles and drones.
"Meriel's work is addressing fundamental challenges for the deployment of Robust autonomous systems," Elbaum said.
Autonomous driving systems, their paper states, require a massive amount of sensed data to train their combined deep neural networks. Yet common sensor hardware migrations are a problem for accurate data flow.
“This necessitates the development of bespoke transformations to adapt new sensor data to the old trained network, or the retraining of a new network with new sensor data.”
These alternatives can be both expensive and impractical, she said.
The paper claims that PreFixer is “the first technique to mitigate the impact of sensor hardware migration in ADS [autonomous driving systems] with a high rate of success, low cost to implement and minimal disruption to the deployment pipeline.”
A Virtual Road Test
Von Stein and her colleagues tested their method for camera sensor migrations on a high-fidelity driving simulator, recreating 10 roadways of 100 meters each that the encoder hadn't seen before. She wanted to see how well the car could maintain its path and how far it could safely drive with the newly augmented sensors, as compared to other available solutions.
But the PreFix augmented encoder, a part of the simulated camera sensors, adds a preprocessing step to the DNN, she explained.
“This encoder has been trained to optimize for reconstruction of the sensor input as well as the DNN prediction for that sensor input.”
Together, the two portions of the tool work like an interpreter for the decoder that feeds the controller. Fewer thoughts get lost in translation. Theoretically, that results in straighter, more accurate lines of travel and fewer stops due to DNN confusion.
How Did PreFixer Perform?
“We measured deviation from the centerline of the road and how many meters on average the car drove with the new sensor and the encoder in place,” von Stein said.
The team’s results showed that their method performed as well or better than other readily available methods of compensating for the loss of a previous camera sensor, while needing only 10% of the usual training data to do so.
The time to train the DNN, which can in some cases take days to weeks of computation time, was cut by 50%.
She said the next steps toward possible industry adoption of PreFix might be for systems undergoing rapid prototyping that need to avoid the arduous process of data collection every time the system undergoes a change in perception hardware.
Discussing her research has been just one aspect of what she said has been a diverse and rewarding Copenhaver Fellowship experience. Among its advantages, the fellowship has allowed von Stein to plan visits to different robotics labs and apply her techniques to different systems, from robotic manipulator arms to medical devices.
Publication Information
M. von Stein, H. Wang and S. Elbaum, "Automated Generation of Transformations to Mitigate Sensor Hardware Migration in ADS," in IEEE Robotics and Automation Letters, vol. 9, no. 7, pp. 6480-6487, July 2024, doi: 10.1109/LRA.2024.3405810.